Maximally informative pairwise interactions in networks 
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Several types of biological networks have recently been shown to be accurately described by 
a maximum entropy model with pairwise interactions, also known as the Ising model. Here we 
present an approach for finding the optimal mappings between input signals and network states 
that allow the network to convey the maximal information about input signals drawn from a given 
distribution. This mapping also produces a set of linear equations for calculating the optimal 
Ising model coupling constants, as well as geometric properties that indicate the applicability of 
the pairwise Ising model. We show that the optimal pairwise interactions are on average zero for 
Gaussian and uniformly distributed inputs, whereas they are non-zero for inputs approximating 
those in natural environments. These non-zero network interactions are predicted to increase in 
strength as the noise in the response functions of each network node increases. This approach also 
suggests ways for how interactions with unmeasured parts of the network can be inferred from the 
parameters of response functions for the measured network nodes. 

PACS numbers; 87.18.Sn, 87.19.11, 87.19. lo, 87.19.1s 
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I. INTRODUCTION 

Many organisms rely on complex biological networks 
both within and between cells to process information 
about their environments P, Q ■ As such, their perfor- 
mance can be quantified using the tools of information 
theory Because these networks often involve 

large numbers of nodes, one might fear that difficult-to- 
measure high-order interactions are important for their 
function. Surprisingly, recent studies have shown that 
neural networks [5, U H ES] , gene regulatory networks 
prl . IT^ . and protein sequences [H, can be accu- 
rately described by a maximum entropy model including 
only up to second-order interactions. In these studies the 
nodes of biological networks are approximated as existing 
in one of a finite number of discrete states at any given 
time. In a gene regulatory network the individual genes 
are binary variables, being either in the inactivated or 
the metabolically expensive activated states. Similarly, 
in a protein the nodes are the amino acid sites on a chain 
which can take on any one of twenty values. 

We will work in the context of neural networks, where 
the neurons communicate by firing voltage pulses com- 
monly referred to as spikes When considered in 
small enough time windows, the state of a network of 
N neurons can be represented by a binary word a = 
(cti, <72, ■ ■ ■ , <^n), where the state of neuron i is given by 
tJi = 1 if it is spiking and = — 1 if it is silent, similar 
to the t/i states of Ising spins. 

The Ising model, developed in statistical physics to 
describe pairwise interactions between spins, can also be 
used to describe the state probabilities Per of a neural 
network: 



Pa = ^cxp 



hi<Ti + ^ Jijaiaj 



(1) 



Here, Z is the partition function and the parameters {hi} 
and {Jij} are the coupling constants. This is the least 
structured (or equivalently, the maximum entropy [1G\ ) 
model consistent with given first- and second-order cor- 
relations, obtained by measuring (ai) and {uiUj), where 
these averages are over the distribution of network states. 

In magnetic systems one seeks the response probabili- 
ties from the coupling constants, but in the case of neural 
networks one seeks to solve the inverse problem of deter- 
mining the coupling constants from measurements of the 
state probabilities. Because this model provides a concise 
and accurate description of response patterns in networks 
of real neurons 0, [1, S [Tot , we are interested in finding 
the values of the coupling constants which allow neural 
responses to convey the maximum amount of information 
about input signals. 

The Shannon mutual information can be written as 
the difference between the so-called response and noise 
entropies [1]: 

The response entropy quantifies the diversity of network 
responses across all possible input signals {T}. For our 
discrete neural system this is given by 



Hr, 



^P,l0g2 (P.). 



(2) 



In the absence of any constraints on the neural responses, 
ifresp is maximized when all 2^ states are equally likely 

The noise entropy takes into account that the network 
states may vary with respect to repeated presentations of 
inputs, which reduces the amount of information trans- 
mitted. The noise entropy is obtained by computing the 
conditional response entropy Pali, and averaging over all 
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inputs, 
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(3) 



where Pj is the input probabiUty distribution. Thus in 
order to find the maximally informative coupling con- 
stants, we must first confront the difficult problem of 
finding the optimal mapping between inputs T and net- 
work states a. 



II. DECISION BOUNDARIES 

The simplest mappings from inputs to neural response 
involve only a single input dimension [H, [l^ |20, HH- 
In such cases, the response of a single neuron can of- 
ten be described by a sigmoidal function with respect to 
the relevant input dimension [H, [22|. However, stud- 
ies in various sensor y sy stems, including the retina [23j . 
the primary visual [M HI, HI, [l^ , auditory and 
somatosensory [2^ cortices, have shown that neural re- 
sponse can be affected by multiple input components, re- 
sulting in highly nonlinear, multi-dimensional mappings 
from input signals to neural responses. 

In Fig. [l]we provide examples of response functions es- 
timated for two neurons in the cat primary visual cortex 
(SQj . For each neuron, the heat map shows the average 
firing rate in the space of the two most relevant input 
dimensions. As this figure illustrates, even in two dimen- 
sions the mapping from inputs to the neural response (in 
this case the presence or absence of a spike in a small 
time bin) can be quite complex. Nevertheless, one can 
delineate regions in the input space where the firing rate 
is above or below its average (red solid lines). As an ap- 
proximation, one can equate all firing rate values to the 
maximum value in regions where it is above average, and 
to zero in regions where it is below average. This ap- 
proximation of a sharp transition region of the response 
function is equivalent to assuming small noise in the re- 
sponse. Across the boundary separating these regions, 
we will assume that the firing rate varies from zero to 
the maximum in a smooth manner (inset in Fig. 

As we discuss below, this approximation simplifies 
the response functions enough to make the optimization 
problem tractable, yet it still allows for a large diversity 
of nonlinear dependencies. Upon discretization into a bi- 
nary variable, the firing rate of a single neuron can be 
described by specifying regions in the input space where 
spiking or silence is nearly always observed. We will as- 
sume that these deterministic regions are connected by 
sigmoidal transition regions called decision boundaries 
[32| . near which P(j\-x ~ 0.5. The crucial component in 
the model is that the sigmoidal transitions are sharp, af- 
fecting only a small portion of the input space. Quanti- 
tatively, decision boundaries are well defined if the width 
of the sigmoidal transition region is much smaller than 
the radius of curvature of the boundary. 
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FIG. 1: (Color online) Example analysis of firing rate for a 
simple, a, and a complex, b, cell in the cat primary visual cor- 
tex probed with natural stimuli from the dataset [s^]. Two 
relevant input dimensions were found for each neuron using 
the maximally informative dimensions method described in 
[Slj . Color shows the firing rate as a function of input simi- 
larity to the first (x-axis) and second (y-axis) relevant input 
dimensions. The values on the x- and y- axis have been nor- 
malized to have zero mean and unit variance. Blue (dashed) 
lines show regions with signal-to-noise ratio > 2.0. Red (solid) 
lines are drawn at half the maximum rate and represent esti- 
mates of the decision boundaries. 



The decision boundary approach is amenable to the 
calculation of mutual information. The contribution to 
the noise entropy -ffnoiso from inputs near the bound- 
ary is on the order of one bit and decays to zero in the 
spiking/silent regions (Fig. We introduce a weight- 
ing factor 77 to denote the summed contribution of in- 
puts near a decision boundary obtained by integrating 



log2 (P„\j.) across the boundary. The factor 



rj depends on the specific functional form of the transi- 
tion from spiking to silence, and represents a measure 
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all decision boundaries weighted by Px, 

N 

^^noisc ~ ds Px, (5) 

where 7„ is the decision boundary of the n"^ neuron. In 
this paper we will assume that t] is the same for all neu- 
rons and is position-independent, but the extension to 
the more general case of spatially varying 77 is possible 
[3^ . Finding the optimal mapping from inputs to net- 
work states can now be turned into a variational problem 
with respect to decision boundaries shapes and locations. 



FIG. 2: (Color online) Schematic of response probability and 
noise entropy. The response function in two dimensions (in- 
set) is assumed to be deterministic everywhere except at the 
transition region which may curve in the input space. In 
a direction x, perpendicular to some point on the decision 
boundary, the response function is sigmoidal (blue, no shad- 
ing) going from silent to spiking. The conditional response en- 
tropy (red, shading underneath) is -Pspiko|x logj (fspiko|x) - 
(1 - -Pspikeix) loga (1 - -Pspikoix) and decays to zero at a; = 
±co. The contribution to the total noise entropy due to this 
cross-section, rj, is the shaded area under the conditional re- 
sponse entropy curve. 



of neuronal noise. In a single-neuron system, the total 
noise entropy is then an integral along the boundary, 
^noiso ~ ?7 J^ds Px, where 7 represents the bound- 
ary, and the response entropy is -ffrcsp = ^-Plog2^' — 
(1 — P) log2 (1 — P), where P is the spike probability of 
the neuron, equal to the integral of Px over the spiking 
region. 

The decision boundary approach is also easily extended 
to the case of multiple neurons, as shown in Fig. [3ja). 
In the multi-neuronal circuit, the various response pat- 
terns are obtained from intersections between decision 
boundaries of individual neurons. In principle, all 2^ 
response patterns can be obtained in this way. We de- 
note as Ga the region of the space where inputs elicit 
a response a from the network. To calculate the re- 
sponse entropy for a given set of decision boundaries in 
a £)— dimensional space, the state probabilities are eval- 
uated as 13— dimensional integrals over {G^} 



P. 



d^rPx (r) 



(4) 



weighted by Px- Just as in the case of a single neuron, 
the network response is assumed to be deterministic ev- 
erywhere except near any of the transition regions. Near 
a decision boundary, the network can be with approxi- 
mately equal probability in one of two states that differ 
in the response of the neuron associated with that bound- 
ary. Thus, such inputs contribute ~ 1 bit to the noise 
entropy, cf. Eq. ^ and Fig. O The total noise entropy 
can therefore be approximated as a surface integral over 



III. RESULTS 
A. General solution for optimal coupling constants 

Our approach for finding the optimal coupling con- 
stants consists of three steps. The first step is to find 
the optimal mapping from inputs to network states, as 
described by decision boundaries. The second step is 
to use this mapping to compute the optimal values of 
the response probabilities by averaging across all possi- 
ble inputs. The final step is to determine the coupling 
constants of the Ising model from the set of optimal re- 
sponse probabilities. 

Due to a high metabolic cost of spiking, we are in- 
terested in finding the optimal mapping from inputs to 
network states that result in a certain average spike prob- 
ability across all neurons: 



1 



(6) 



where Sa is the number of "up spins" , or firing neurons, 
in configuration a. Taking metabolic constraints into 
account, we maximize the functional 



F = H,, 



X\ NP 



N 



P. 



E 



p. - J d'^rPx (r) j 



(7) 



where A, {(3a-} are Lagrange multipliers for the con- 
straints and the last term demands self-consistency 
through Eq. (g]) . 

To accomplish the first step, we optimize the shape of 
each segment between two intersection points. Requiring 
^ = yields the following equation 

77 [k + n • VPx (r)] + ^<Ti,...,CT._i,T,ai+i,...,<T« 

~'/5o'l,...,(Ti-l,J,,(Ti+l,...,(Tjv (^) 

for the segment of the i*'' decision boundary that 
separates the regions G'cri,...,CTi_i,T,(Ti+i,...,CTjv a-nd 
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FIG. 3: (Color online) Network decision boundaries. Color in 
input space corresponds to the input distribution Px and each 
line is a boundary for an individual neuron, a, For a general 
intersection between neurons i and j, the boundaries divide 
the space into four regions corresponding to the possible net- 
work states (e.g. 00-^=1, ctj=|). Each segment has a width rj 
which determines the noise level, and is described by a param- 
eter A^V, cf. Eq. (I10|l . b, In a uniform space, with two neu- 
rons (different colors/shading), the boundary segments satis- 
fying the optimality condition are circular. Networks which 
have segments with different curvatures (dashed) are less in- 
formative than smooth circles (solid), c. In a Gaussian space 
(in units of standard deviations), straight perpendicular lines 
(solid) provide more information about inputs than decision 
boundaries that intersect at any other angle (dashed), d, For 
approximately natural inputs (plotted in units of standard de- 
viations), the suboptimal balanced solutions (dashed) are two 
independent boundaries with the same P. The optimal solu- 
tions (solid) change their curvature at the intersection point, 
and depend on the neuronal noise level rj. 



Gai (Ti_i,x,CTi+i,...,(TN- Here, n is the unit normal 

vector to the decision boundary, and k = V • n is the 
total curvature of the boundary. We then optimize with 
respect to the state probabihties, jp- = 0, which gives a 
set of equations 



f3a — •S'tjA 



ln2 



(1 + InP.) 



(9) 



Combining Eqs. ([8]) and ([9]), we arrive at the follow- 
ing equation for the segment of the i*'* decision boundary 
across which ai changes while leaving the rest of the net- 
work in state a' : 



A^? + K + n • VPx (r) = 0. 



The parameter 



A — loa 



■ {T-l,...,ai-l,\ ,(Ti+-l,...,ITN 
P 

CTi ,4,,cri-|-i,...,crjv 



(10) 



(11) 



is specific to that segment, and is determined by the 
ratio of probabilities of observing the states which this 
segment separates. Generally, this ratio (and therefore 
the parameter A^V) may change when the boundaries 
intersect. For example, in the schematic in Fig. [HJa), 

depends on the ratio of Pai=^^aj=i/Pcr,=i,crj=i, 
whereas A[,*j^-|. depends on P^i^i^aj^'i /Pai=i,aj=p The 
values of the parameters for the two segments of the i*'' 
boundary are equal only when Pa-i=i.a.i=i/Pa-i=i,crj=i — 
Pai='\,aj='\ / PcTi=i,crj='[- This Condition is satisfied when 
the neurons are independent. Therefore, we will refer to 
the special case of a solution where A does not change 
its value across any intersection points as an independent 
boundary. In fact, Eq. (llOp with a constant A is the 
same as was obtained in ^3^] for a network with only one 
neuron. In that case, the boundary was described by a 
single parameter A, which was determined from the neu- 
ron's firing rate. Thus, in the case of multiple neurons, 
the individual decision boundaries are concatenations of 
segments of optimal boundaries computed for single neu- 
rons with, in general different, constraints. A change 
in A^V across an intersection point results in a kink - an 
abrupt change in the curvature of the boundary. Thus, by 
measuring the change in curvature of a decision bound- 
ary of an individual neuron one can obtain indirect mea- 
surements on the degree of interdependence with other, 
possibly unmeasured, neurons. 

Our main observation is that the A-parameters deter- 
mining decision boundary segments can be directly re- 
lated to the coupling constants of the Ising model through 
a set of linear relationships. For example, consider two 
neurons i and j within a network of N neurons whose de- 
cision boundaries intersect. It follows from Eq. pT|) that 
the change in A-parameters along a decision boundary is 
the same for the ith and jth neuron, and is given by: 



A 



1 



A 



logs 



(0 

Pn" 



= A 



0) 

<Ti = T " 



-A 

Pa" 



<Ti = i 



-Pct " , (T i = t , (T j = J. -Per " , (T i = X , £7 j = I 



(12) 



where a" represents the network state of all the neurons 
other than i and j. Taking into account the Ising model 
via Eq. ([1]), this leads to a simple relationship for the 
interaction terms {Jij}'. 



A 



(i) 

CTj=T 



-A 



2 {Jjj + Jji) 
7yln2 



(13) 



We note that only the average ( "symmetric" ) compo- 
nent of pairwise interactions can be determined in the 
Ising model. Indeed, simultaneously increasing Jy and 
decreasing Jji by the same amount will leave the Ising 
model probabilities unchanged because of the perturba- 
tion symmetry in Eq. ([T]) . This same limitation is present 
in the determination of {Jij} via any method (e.g. an in- 
verse Ising algorithm). 
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Once the interaction terms {Jij} are known, the local 
fields can be found as well from Eq. pT|) : 



r;ln2 



(14) 



This equation can be evaluated for any response pattern 
a' , because consistency between changes in A is guaran- 
teed by Eq. p^. 

The linear relationships between the Ising model cou- 
pling constants and the {X^^}} parameters are useful, 
because they can indicate what configurations of net- 
work decision boundaries can be consistent with an Ising 
model. First, Eq. (jl3p tells us that if the i"^ boundary 
is smooth at an intersection with another boundary j, 
then the average pairwise interaction in the Ising model 
between neurons i and j is zero [as mentioned above the 
cases of truly zero interaction = Jji — and that of a 
balanced coupling Jy = —Jji cannot be distinguished in 
an Ising model]. Second, we know that if one boundary 
is smooth at an intersection, then any other boundary it 
intersects with is also smooth at that point. More gen- 
erally, the change in curvature has to be the same for 
the two boundaries and we can use it to determine the 
average pairwise interaction between the two neurons. 
A third point is that the change in curvature has to be 
the same at all points where the same two boundaries 
intersect. For example, intersection between two planar 
boundaries is allowed because the change is curvature is 
zero at points across the intersection line. In cases where 
intersections form disjoint sets the equal change in cur- 
vature would presumably have to be due to a symmetry 
of decision boundaries. 

In summary then, we have the analytical equations for 
the maximally informative decision boundaries of a net- 
work, through Eqs. pO|) and (fTTj) . We now study their 
solutions for specific input distributions and then deter- 
mine, through Eqs. and p4p . the corresponding 
maximally informative coupling constants. 



B. Uniform and Gaussian distributions 

We first consider the cases of uniformly and Gaussian 
distributed input signals. As discussed above, finding op- 
timal configurations of decision boundaries is a trade-off 
between maximizing the response entropy and minimiz- 
ing the noise entropy. Segments of decision boundaries 
described by Eq. (jlOp minimize the noise entropy lo- 
cally, whereas changes in the A parameters arise as a 
result of maximizing the response entropy. The inde- 
pendent decision boundaries, which have one constant 
A for each boundary, minimize the noise entropy glob- 
ally for a given firing rate. Because for one neuron, 
specifying the spike probability is sufficient to deter- 
mine the response entropy Pi ~ P, cf. Eq. ([S]), and 
^^rcsp = — -Plog2 P~ (1 ~ P) log2 (1 — P), maximizing in- 
formation for a given spike probability is equivalent to 
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FIG. 4: (Color online) Information from independent and op- 
timally coupled network boundaries as a function of neuronal 
noise level. In both examples shown here, the independent 
boundaries (blue circles) lose information faster than the op- 
timal boundaries (red triangles) as the noise level rj increases. 
Each curve represents two response probabilities (e.g. P = 
0.45 and 0.55) because information is invariant under switch- 
ing the spiking/silent regions. 



minimizing the noise entropy [32] . When finding the op- 
timal configuration of boundaries in a network with an 
arbitrary number of nodes, the response entropy is not 
fixed, because the response probability may vary for each 
node (it is specified only on average across the network). 
However, if there is some way of arranging a collection 
of independent boundaries to obtain response probabili- 
ties that also maximize the response entropy, then such a 
configuration must be optimal because it simultaneously 
minimizes the noise entropy and maximizes the response 
entropy. It turns out that such solutions are possible 
for both the uniformly and Gaussian distributed input 
signals. 

For the simple case of two neurons receiving a two- 
dimensional uniformly distributed input, as in Fig. [Sljb), 
the optimal independent boundaries are circles, because 
they minimize the noise entropy (circumference) for a 
given probability P (area). In general, the response en- 
tropy, Eq. ([2]) , is maximized for the case of two neurons 
when the probability of both neurons spiking is equal to 
P^. It is always possible to arrange two circular bound- 
aries to satisfy this requirement. The same reasoning ex- 
tends to overlapping hyperspheres in higher dimensions, 
allowing one to calculate the optimal network decision 
boundary configuration for uniform inputs. Therefore, 
for uniformly distributed inputs the optimal network de- 
cision boundaries are overlapping circles in two dimen- 
sions or hyperspheres in higher dimensions. 

For an uncorrelated Gaussian distribution. Fig. [Sljc), 
the independent boundaries are {D — l)-dimensional hy- 
perplanes [32| . If we again consider two neurons in a two- 
dimensional input space, then the individual response 
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probabilities for each boundary determine the perpen- 
dicular distance from the origin to the lines. Any two 
straight lines will have the same noise entropy, indepen- 
dent of the angle between them. However, orthogonal 
lines (orthogonal hyperplanes in higher dimensions) max- 
imize the response entropy. The optimality of orthogonal 
boundaries holds for any number of neurons and any in- 
put dimensionality. We also find that for a given average 
firing rate across the network, the maximal information 
per neuron does not depend on input dimensionality as 
long as the number of neurons N is less than the input 
dimensionality D. For N > D, the information per neu- 
ron begins to decrease with N, indicating redundancy 
between neurons. 



C. Naturalistic distributions 

Biological organisms rarely experience uniform or 
Gaussian input distributions in their natural environ- 
ments and might be evolutionarily optimized to opti- 
mally process inputs with very different statistics. To 
approximate natural inputs, we use a two-dimensional 
Laplace distribution. Pi (r) oc exp (— |x| — which 
captures the large-amplitude fluctuations observed in 
natural stimuli [33l |3J], as well as bursting in protein 
concentrations within a cell f35'| . For this input distribu- 
tion there are four families of solutions to Eq. (fTO)) (see 
[s^ l for details), giving rise to many potentially optimal 
network boundaries. For a given A, the decision bound- 
aries can be found analytically. To find the appropriate 
value of the AV, we numerically solved Eq. (10) using 
Mathematica [3a|. We found no solutions for indepen- 
dent boundaries. The optimal boundaries therefore will 
have different A's, and kinks at intersection points. As a 
result, the neurons will have a nonzero average coupling 
between them, examples of which are shown in Fig. [3Kd). 

We found that the shapes of the boundaries change 
with the noise level 77, which does not happen for inde- 
pendent boundaries [33 • To see if this noise dependence 
gives the network the ability to compensate for noise in 
some way, we look at the maximum possible information 
the optimal network boundaries are able to encode about 
this particular input distribution for different noise levels 
(Fig. |4]). We compare this to the suboptimal combination 
of two independent boundaries with the same P. The 
figure illustrates that the optimal solutions decrease in 
information less quickly as the noise level increases. The 
improvement in performance results from their ability to 
change shape in order to compensate for the increasing 
noise level. 

We calculated both h = hi = and J = 
(J12 + J21) /2 for various noise levels and response prob- 
abilities. Fig. [Sfa) and (c) show the local field h is prac- 
tically independent of the noise level but does depend on 
the response probability. The coupling strength, how- 
ever, depends on both noise level and response probabil- 
ity, increasing in magnitude with neuronal noise, shown 
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FIG. 5: (Color online) Optimal coupling constants for natu- 
ralistic inputs, a. The local fields show very little dependence 
on the noise level of the neurons, but the magnitude of the in- 
teraction strength J, b, increases in the same fashion with the 
noise level regardless of the response probability P. c, h de- 
pends strongly on the response probability, d, J changes sign 
about P = 1/2. Below this point the coupling is excitatory, 
and above it the coupling is inhibitory. 



in Fig. [S]Jb). The combination of this result and the 
noise compensation observed in Fig. IH suggests that the 
network is able to use pairwise coupling for error correc- 
tion, with larger noise requiring larger coupling strengths. 
This strategy is similar to the adage "two heads are bet- 
ter than one" , to which we add ". . . especially if the two 
heads are noisy" . 

In Fig. [3d) we observe that the sign of the couphng 
changes as the value of P crosses 1/2. When P — 1/2, 
the optimal solution is an X crossing through the origin, 
which is the only response probability for which the net- 
work boundary is made of two independent boundaries, 
making J = for any noise level when P = 1/2. It 
can also be seen that J ^ as 77 ~> for P ^ 1/2. 
Curiously, for a given rj, the dependence of the optimal 
J on P is highly nonmonotonic: it changes sign across 
P = 1/2, and reaches a maximum (minimum) value for 
P w 0.25(0.75). 



IV. DISCUSSION 

The general mapping between inputs and maximally 
informative network responses in the decision boundary 
approximation has allowed us to calculate the Ising model 
coupling constants. In this approach, network responses 
to a given input are not described by an Ising model, 
which emerges after averaging network responses across 
many different inputs. Although there are many con- 
figurations of network decision boundaries that can be 
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consistent with a pairwise-interaction model, certain re- 
strictions apply. For example, the change in curvature of 
a decision boundary that can occur when two boundaries 
intersect has to be the same for both boundaries, and, if 
they intersect at more than one disjoint surfaces then the 
curvatures of those surfaces must be the same. 

We find that for both the uniform and Gaussian in- 
put distributions the optimal network boundaries are 
independent. This implies that the average interaction 
strength is zero for all pairs of nodes through Eq. 
Such balance between excitatory and inhibitory connec- 
tions has been observed in other contexts including rapid 
state switching fSTj, Hebbian learning [sl], and selective 
amplification [39]. In this context balanced coupling is 
just one possible configuration of a network of decision 
boundaries and happens to be optimal for uniformly and 
Gaussian distributed inputs. 

For a more realistic input distribution, the Laplace 
distribution, we found the optimal boundaries were not 
smooth at intersection points. This indicates that the av- 
erage coupling between the nodes in the network should 
be non-zero to achieve maximal information transmis- 
sion. We also observed that the optimal configuration of 
the network depended on the noise level in the responses 
of the nodes, giving the network the ability to partially 
compensate for the encoding errors introduced by the 
noise, which did not happen for the less natural input 
distributions considered. Also, the fact that J can be 
positive or negative between two nodes leads to the po- 
tential for many stable states in the network, which could 
give the network the capacity to function as autoassocia- 
tive memory, as in the Hopfield model [13, |4l| . Similar 
network behaviors were reported in [4^ for networks of 
ten neurons, where the optimal coupling constants were 
numerically found for correlated binary and Gaussian in- 
puts. Our approach is different in that we use an Ising 
model to describe average network responses, but not re- 
sponses to particular inputs, Pa\x- 

Previous experiments have shown that simultaneous 
recordings from neural populations could be well de- 
scribed by the Ising model. In one such experiment us- 
ing natural inputs [7j , the distributions of coupling con- 
stants showed an average h which was of order unity and 
negative and average J which was small and positive. 
Our results for the Laplace distribution are in qualitative 
agreement with these previous findings if one assumes a 
response probability P < 1/2. Due to the high metabolic 
cost of spiking, this is a plausible assumption to make. 

The method we have put forth goes beyond predicting 
the maximally informative coupling constants, to make 
statements about optimal coding strategies for networks. 
Although both uniform and Gaussian inputs can be opti- 
mally encoded by balanced networks, for example, their 
organizational strategies are remarkably different. In the 
uniform input case, the optimal boundaries curve in all 
dimensions, meaning each node is attending to and en- 
coding information about every component of the possi- 
bly high-dimensional input, and they organize themselves 



by determining the optimal amount of overlap between 
boundaries. However, for the Gaussian distribution, each 
boundary is planar, indicating that the nodes of the net- 
work are sensitive to only one component of the input. 
The optimal strategy for networks receiving this type of 
input is to attend to and encode orthogonal directions 
in the input space, minimizing the redundancy in the 
coding process. 

In terms of practical applications, perhaps the most 
useful aspect of this framework is the ability to infer 
the strength of pairwise interactions with other nodes in 
the network by examining decision boundaries of single 
nodes, cf. Eq. P^ . 

The observation of different types of pairwise interac- 
tions for networks processing Gaussian and naturalistic 
Laplace inputs raises the possibility of discovering novel 
adaptive phenomena. Previous studies in several sensory 
modalities have demonstrated that the principle of infor- 
mation maximization can account well for changes in the 



48, 
54, 



55| 



relevant input dimensions [30|, |43|, IM |45|, 143, [47 
[sol , [sil . m, Issj as well as the neural gain 22 , 52 
following changes in the input distribution. For example, 
nonlinear gain functions have been shown to rescale with 
changes in input variance [1^, HI, [H, [sl, [s^. Our re- 
sults suggest that if neurons were adapted on one hand to 
Gaussian inputs, and then to naturalistic non-Gaussian 
inputs, then multi-dimensional input /output functions 
of individual neurons might change qualitatively, with 
larger changes expected for noisier neurons. 

By studying the geometry of interacting decision 
boundaries we have gained insights into optimal coding 
strategies and coupling mechanisms in networks. Our 
work focused on the application to neural networks, but 
the method developed here is general to any network with 
nodes which have multidimensional, sigmoidal response 
functions. Although we have only considered three par- 
ticular distributions of inputs, the framework described 
here is general and can be applied to other classes of in- 
puts with the potential of uncovering novel, metabolically 
efficient combinatorial coding schemes. In addition to 
making predictions for how optimal pairwise interactions 
should change with adaptation to different statistics of 
inputs, this approach provides a way to infer interactions 
with unmeasured parts of the network simply by observ- 
ing the geometric properties of decisions boundaries of 
individual neurons. 
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